In the specification 

In accordance with 37 CFR §1.125, a clean version of the substitute specification 
(pages 1- ) is provided below, and a marked up version of the specification is also 
attached hereto. 

Delete the paragraphs beginning at page L line 1 th rough page 33 line 5 and 
replace them with the following: 

TETHOD AND WsTEM FOR TESTING SOFTWARE COMPONENTS 
This invention relates generally to computer software applications and more 
specifically to testingVcomputer software applications. 

Distributed coirbuting has been used for many years. Distributed computing is 
very prevalently used iA "enterprise-wide" applications. An enterprise-wide application 
is an application that alloVs a large group of people to work together on a common task. 
Usually, an enterprise-wide application performs functions that are essential to a 
company's business. For Lample, in a bank, people at every bank branch must be able 
to access a database of accounts for every bank customer. Likewise, at an insurance 
company, people all over thk company must be able to access a database containing 
information about every policVolder. The software that performs these fimctions is 
generally knovm as enterprise-wiMe applications. 

As available hardware and\oftware has evolved, the architecture of enterprise 
wide applications has changed. An ikrchitecture which is currently popular is called the 
N-Tier enterprise model. The mostWvalent N-tier enterprise model is a three tier 
model. The three tiers are the fi-ont end,\he middleware and the back end. The back end 
is the database. The front end is sometimes referred to as a "client" or a Graphical User 
Interface (GUI). The middleware is the ^bftware that manages interactions with the 
database and captures the "business logic."\ Business logic tells the system how to 
validate, process and report on the data in a fa^ion that is usefiil for the people in the 
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w^terprise. 

The middleware resides on a computer called a server. The database might be on 
the same compvyt or a different computer. The "client" is usually on an individual's 
personal compute^\\ll of the computers are connected together through a network. 
Because many peopl\ use the enterprise wide application, such systems are set up to 
allow simultaneous usfiffs and there would be many clients connected to a single server. 
Often, many clients will\be connected to the server simultaneously. 

Those familiar with Internet commerce will recognize that the N-tiered model 
also describes many Inteme\ web sites that sell goods or services. For example, a web 
site that auctions cars is likelAto fit the N-tiered model. In such an application, databases 
are provided to track buyers, sMlers and objects being auctioned. Also, a database must 
be provided to track the bids as they are entered. The middleware provides the access to 
these databases and encapsulates the business logic around such transactions as when to 
accept a bid, when to declare an it^ sold, etc. In the world of distributed computing, it 
makes no difference whether the "cliWs" using the application are employees of a single 
company or many Internet users throitehout the world. Herein, examples of applications 
under test will be given, but they are n\)t intended to imply limitations on the use of the 
invention. The inventions described hetein could be used by developers of enterprise- 
wide applications or web based applicationfe. 

One advancement in the N-tiered mod\l is that the middleware is very likely to be 
componentized and is very likely to be writtefl to a component standard so that it will 
easily integrate with software at other tiers. Entaprise JavaBeans (EJB technology based 
software components) by Sun Microsystems, COM, DCOM and COM+ by Microsoft 
Corporation and CORBA by IBM are examples if component specification standards 
that are commercially available. Herein, EJB techriology based software component is 
used as an example of a component standard used ti) implement middleware in an N- 
tiered model , but it should be appreciated that and the excepts described herein could be 
used with other component standards. 
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/ EJB technology based software components are written in the JAVA language, 
Which is intended to be "platform independent." Platfonn independent means that an 
application is intended to perform the same regardless of the hardware and operating 
system on which ik is operating. Platform independence is achieved through the use of a 
"container." A container is software that is designed for a specific platform. It provides 
a standardized envirWnent that ensures the application written in the platform 
independent language Vperates correctly. The container is usually commercially 
available software and the\application developer will buy the container rather create it. 

Componentized software is software that is designed to allow different pieces of 
the application, or "objects", to be created separately but still to have the objects work 
together. For this to happen, W objects must have standard interfaces that can be 
understood and accessed by other bbjects. Some parts of these interfaces are enforced by 
the software language. If the interfabes are not used, the software objects will not be able 
to work with other objects. Other ]*actices are imposed by convention. Usually, one 
company has "control" over the lanVuage and specifies programming practices that 
should be followed by anyone writing tolatform independent software in that language. 
Because these programming practices ar^known to everyone, the companies that create 
the containers can rely on them when cheating the container. As a result, if these 
practices are not followed, the container mteht not operate properly. Thus, there is an 
indirect mechanism for enforcing these practices. 


Typically, applications have been testedVn one of two ways. The objects are 
tested as they are written. Each is tested to ensureWt it performs the intended ftmction. 
When the objects are assembled into a completed Application, the entire application is 
then usually tested. Heretofore, application testing hlas generally been done by applying 
test inputs at the client end and observing the response of the application. There are 
several shortcomings with this process. One is that\it is relatively labor intensive, 
particularly to develop a load or scalability test. There Vas been no easy way to create 
the test program, instantiate it with test data, execute the te« and aggregate the results. 
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SoAe tools, called "profilers," have been available. However, these tools track 
things such as disk usage, memory usage or thread usage of the application under test. 
They do not provide data about performance of the application based on load. 


Other toolsyare available to automate the execution of tests on applications. For 
example, RSW SoftWare, Inc. of Waltham, MA, provides a product called e-Load. This 
tool simulates load on an application under test and provides information about the 
performance of the application. However, this tool does not provide information about 
the components in an application. We have recognized that a software developer would 
find such information veryusefiil. 


Automatic test generation tools, such as TestMaster available from Teradyne 
Software and System Test of Nafehua, NH, are also available. Tools of this type provide a 
means to reduce the manual effoV of generating a test. TestMaster works fi'oni a state 
model of the application under tesK Such an application is very usefiil for generating 
functional tests during the development of an application. Once the model of the 
application is specified, TestMaster cdn be instructed to generate a suite of tests that can 
be tailored for a particular task - such akto fully exercise some portion of the application 
that has been changed. Model based testing is particularly useful for functional testing of 
large applications, but is not fiiUy automatic because it requires the creation of a state 
model of the application being tested. 

We have recognized that a second shortcoming of testing enterprise wide 
applications is the critical performance criterik to measure often relates to how the 
application behaves as the number of simultaneous users increases. There are examples 
of websites crashing or operating so slow as to frustrate an ordinary user when too many 
users log on simultaneously. In the past, load has been simulated informally, such as by 
having several people try to use the application at the same time. Some tools exist to 
provide a load on an application for testing, such k e-Load available fi-om RSW of 
Waltham, MA. 
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However, it has generally not been until the application is deployed into its 
intended operating environment that the performance of the application under load is 
known. ThusVhe biggest problem facing an application developer might not be testing to 
see whether eact object performs as designed or even whether the objects work together 
as a system. HeWofore there has been no available tool that will help an application 
developer ascertairi how many simultaneous users a middleware application can 
accommodate given k specified transaction response time or identify which object in the 
application, given real world load conditions, is causing the bottleneck. SUMMARY OF 
THE INVENTION 


With the foregoing background in mind, it is an object of the invention to provide 
testing tools to facilitate load ttased testing of N-tiered applications. 

It is also an object to provide automatic testing. 

The foregoing and other objkts are achieved by a test system that generates test 
code for components of an application under test using interface information about the 
components. 

In the preferred embodiment, the generated test code simulates use of a particular 
software object within an application by a plurality of simultaneous users. The number 
of simultaneous users simulated is varied. 


In a presently preferred embodiments, the test code is generated from interface 
information for the object under test, which identifies methods of the object. Ranges and 
formats for data values for the inputs and outputs W the methods are determined from 
properties specified in the interface information. Specific data values are generated from 
user supplied tables or using algorithms to pick allowable values within the range. 
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BRIEF DESCRIPTION OF THE DRAWINGS 


The in\^ntion will be better understood by reference to the following more 
detailed descriptiW and accompanying drawings in which 

FIG. 1 is an illustration of an application under test by the test system of the 
inveruion; 

FIG. 2 is an illWration showing the test system of the invention in greater detail; 
FIG. 3 is an illuWtion showing the coordinator of FIG. 2 in greater detail; 
FIG. 4 is a flow\hart illustrating the process of coordinating execution of load 
tests; 

FIG. 5 is an illustrati^ showing the code generator of FIG. 2 in greater detail; 
FIG. 6 illustrates a useXinterface of test system 1 10 during a setup phase; 
FIG. 7 illustrates a user\iterface of test system 110 during the specification of a 
test case; 

FIG. 8 illustrates a user int^ace of test system 110 during a different action as 

part of the specificatiomof a test case; 
FIG. 9 illustrates a user interfa^of test system 110 during a different action as 

part of the specification of 2^ test case; 
FIG. 10 illustrates a user interface ^test system 110 during a different action as 

part of the specification of a tek case; 
FIG. 1 1 illustrates a user interface of te^ system 1 10 during a phase for reviewing 

the results of a test case in tabular Yormat; 
FIG. 12 illustrates a user interface of test system 1 10 during a phase for reviewing 

the results of a test case in graphical fWiat; 
FIG. 13 illustrates a user interface of test systerti 1 10 during a phase for reviewing 

the resuhs of a test case in an alternative graphical format; and 
FIG. 14 illustrates a user interface of test system l\o during a phase for reviewing 

the results of a test case in a tabular format. 
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"^DESCWpVoN OF THE PREFERRED EMBODIMENT 

FIG. \ illustrates a test system 110 according to the present invention. The 
system is testW application under test 114. Here appUcation under test 114 is an 
application in fee N-tiered model. More specifically, it is a three tiered database 
application. Application under test 114 could represent a database for a bank or an 
insurance companyVr it could represent an Internet application. The specific function of 
application under tesKl 14 are not important to the invention. 

Also, the specifiX hardware on which test system 110 and the application vinder 
test 114 reside is not irr^ortant to the invention. It is sufficient if there is some 
connection between the tw(\ In the illustration, that connection is provided by network 
122. In the illustrated embodnnent, it is contemplated that network 122 is part of a LAN 
operated by the owner of application under test 1 14, such as an Ethernet network. In this 
scenario, test system 1 10 is installed on a server within the LAN. However, many other 
implementations are possible. For example, network 122 could be a WAN ovmed by the 
owner of application imder test 114. \ 

A further variation is that network 122 could the Internet. In that scenario, test 
system 1 10 could be located in a server owr^d by a testing company. 

A ftirther possibility is that test system W application under test could be located 
in computers owned by a testing company. MaiV applications are written using platform 
independent technology such that the application will perform the same on many 
different platforms. Platform independent technolcJky is intended to make it easy to run 
an application on any platform. Therefore, the application under test could be sent to a 
testing company, owning the hardware to implemeVt test system 110, such as by 
uploading over the Internet. Thereafter, the applicatio\ under test could be tested as 
described herein while rimning on a platform provided by\the testing company with the 
results of the test being downloaded over the Internet. \ 
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/ Still Oither variations are possible. Test system 1 10 and application under test 1 14 
\ could physicallV be implemented in the same computer. However, that implementation is 
not presently preferred because a single computer would have to be very large or would 
be limited in the Sdze of applications that could be tested. The presently preferred 
embodiment uses sevVal computers to implement test system 110. 

Application unde\ test 114 is a software application as known in the art. It 
includes middleware 116 mat encapsulates some business logic. A user accesses the 
application through a client device. Many types of client devices are possible, with the 
list growing as networks become more prevalent. Personal computers, telephone systems 
and even household appliances with micro- controllers could be the client device. For 
simplicity, the client device is illustrated herein as a personal computer (PC) 120, though 
the specific type of client device is riot important to the invention. PC 120 is connected 
to network 122 and can therefore access application under test 114. In use, it is 
contemplated that there would be multiple users connected to application under test 1 14, 
but only one user is shown for simplicity .VThe number of users simultaneously accessing 
application under test 1 14 is one indication of the "load" on the application. 

Access to the application under test is^ in the illustrated embodiment, through 
Graphical User Interface (GUI) 124 of the type laiown in the art. Software to manage 
interactions between multiple users and an application is known. Such software is 
sometimes called a web server. Web servers opemte in conjunction with a browser, 
which is software commonly foimd on most PC's. \ 

The web server and browser exchange informations a standard format known as 
HTML. An HTML file contains tags, with specific information associated with each tag. 
The tag signals to the browser a type associated with the infWiation, which allows the 
browser to display the information in an appropriate format. F6r example, the tag might 
signal whether the information is a title for the page or whether tW information is a link 
to another web page. The browser creates a screen display m\ particular window 
running on PC 120 based on one or more HTML pages sent by the weti server. 
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^ / Whek a user inputs commands or data into the window of the browser, the 
^ browser uses inforaiation on the HTML page to format this information and send it to 

the web server. Nin this way, the web server knows how to process the commands and 
data that comes from the user. 



GUI 124 passes^he information and commands it receives on to middleware 1 16. 
In the example of FIg\i, middleware 116 is depicted as a middleware application 
created with EJB technology based software components. Containers 130 are, in a 
preferred embodiment, conmiercially available containers. Within a container, are 
numerous enterprise Java beansyl 32. Each Java bean can more generally be thought of as 
a component. GUI 124, baseci on the information from user PC 120, passes the 
information to the appropriate EJBi technology based software component 132. Outputs 
from application under test 1 14 are p\)vided back through GUI 124 to PC 120 for display 
to a user. 

EJB technology based software component's 132, in the illustrated example, 
collectively implement a database application. EJB technology based software 
component's 132 manage interactions with ana\process data from databases 126 and 128. 
They will perform such database fimctions as letting values in a particular record or 
getting values from a particular record. Other functions are creating rows in the database 
and finding rows in the database. EJB technolo^ based software component's that 
access the database are often referred to as "entity bea 


Other types of EJB technology based software component's perform computation 
or control ftmctions. These are called "session beans." \Session beans perform such 
fimctions as validating data entries or reporting to a user\hat an entry is erroneous. 
Session beans generally call entity beans to perform database acVess. 


It will be appreciated that, while it is generally preferable to segregate 
programming of the application in such a way that each type of database transaction is 
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controll^ by a single bean that performs only that function, some entity beans will 
perform functions not strictly tied to database access. Likewise, some session beans will 
perform database access functions without calling an entity bean. Thus, while different 
testing techniques will be described herein for testing session beans and entity beans, it is 
possible that some EJB technology based software component's will have attributes of 
both entity and session beans. Consequently, a full test of any bean might employ 
techniques of testing^ntity beans and testing session beans. 

Test system 110 ikable to access the EJB technology based software component's 
132 of application under ffest 114 over network 122. In this way, each bean can be 
exercised for testing. In the Veferred embodiment, the tests are predominately directed 
at determining the response tone of the beans - or more generally determining the 
response time of components ok objects used to create the application under test. 
Knowing the response time of a beM can allow conclusions about the performance of an 
application. The details of test system\lO are described below. 

In the illustrated embodiment, testNsystem 110 is software installed on one or 
more servers. It is conceptually much like application under test 114. In a preferred 
embodiment, test system 1 10 is a JAVA applicafion. Like application under test 1 14, test 
system 1 10 is controlled through a graphical user iWerface 150. GUI 150 might be a web 
server as known in the art. One or more application, developers or test engineers might 
access test system over the network 122. In FIG. 1, pVs 152 and 154 are PC's used by 
testers who will control the test process. \ 

Like application under test 114, multiple individualsSjnight use test system 110 
simultaneously. For example, multiple testers might be testing ^single application. Each 
tester might be focused on testing different aspects of the applicatrtm. Alternatively, each 
tester might be testing a different application. Numerous applicatioHs might be installed 
on computers on network 122. Test system 1 10 might be testing diffeW applications at 
the same time. \ 
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Tufaiing now to FIG. 2, details of test system 110 are shown. Test system 110 
performs several functions. One function is the generation of test code. A second^ 
function is t\ execute the test code to exercise one or more EJB technology based 
software component's in the application under test. Another function is to record and 
analyze the results of executing the test code. These functions are performed by software 
running on one onmore computers connected to network 122. The software is written 
using a conunercialw available language to perform the functions described herein. 

FIG. 2 shows that test system 110 has a distributed architecture. Software 
components are installedVn several different computers. Multiple computers are used 
both to provide capability ft^r multiple users, to a allow a user to perform multiple tasks 
and also to run very large testk The specific number of computers and the distribution of 
software components of the test system on those computers is not important to the 
invention. 

Coordinator 210 is a software application that interfaces with GUI 150. The main 
purpose of coordinator 210 is to route U^er requests to an appropriate server in a fashion 
that is transparent to a user. Turning tds^FIG. 3, coordinator 210 is shown in greater 
detail. It should be appreciated, though, tKlat FIG. 3 shows the conceptual structure of 
coordinator 210. Coordinator 210 might not\e a single, separately identified piece of 
software. It might, for example, be implemented as coordination software within the 
various other components of test system 110. Also, it should be realized that a web 
server used to implement GUI 150 also provides coordination functions, such as queuing 
multiple requests from an individual or coordinating mWtiple users. 

Coordinator 210 contains distribution unit 312. Distribution unit 312 is preferably 
a software program running on a server. As user requests are received from GUI 150, 
they are received by distribution unit 312. As the requests are\eceived, distribution unit 
312 determines the type of resource needed to process the request. For example, a 
request to generate code must be sent to a server that is running a coo^^generator. 
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/ QDordinator 210 includes several queues to hold the pending requests. Each 
queue is implemented in the memory of the server implementing coordinator 210. In 
FIG. 3, queues 318A...318C are illustrated. Each queue 318A...318C corresponds to a 
particular typeXof resource. For example, queue 318A could contain code generator 
requests, queue Jd8B could contain test engine requests and queue 318C could contain 
data analysis reqiiests. Distribution unit sends each request to one of the queues 
3 1 8 A. . .3 1 8C, based \n the type of resources needed to process the request. 

Associated with each queue 318A...318C is queue manager 320A...320C. Each 
queue manager is preferably implemented as software nmning on the server 
implementing coordinator or the server implementing the relevant piece of 
coordinator 210. Each queue manager maintains a list of servers within test system 1 10 
that can respond to the requestsVi its associated queue. A queue manager sends the 
request at the top of the queue to \ server that is equipped to handle the request. The 
cormection between the queue managV and the servers equipped to handle the requests is 
over network 122. If there are otherNservers available and still more requests in the 
queue, the queue manager will send the rtext request in the queue to an available server. 
When there are no available servers, each queue manager waits for one of the servers to 
complete the processing of its assigned request. 

As the requests are processed, the servers\such as the code generators and the test 
engines report back to the queue managers. lAresponse, the queue managers send 
another request from the queue and also provide thk resuhs back to the distribution unit 
312. Distribution unit 312 can then reply back tAthe user that issued the request, 
indicating that the request was completed and eitherViving the results or giving the 
location of the resuhs. For example, after test code is gWrated, the user might receive 
an indication of where the test code is stored. After a teVt is executed, the user might 
receive a report of the average execution time for the test omhe location of a file storing 
each measurement made during the test. \ 

It will be appreciated by one of skill in the art that softw^e systems that process 
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user commands, including commands from multiple users, are well known. Such 
systemsViust have an interface for receiving commands from a user, processing those 
command^ and presenting results to the user. Such interfaces also allow those results to 
be used byttie user for implementing further commands. Such an interface is employed 
here as well Vid is depicted generally as GUI 150. For example, GUI 150 will allow a 
user to enter command that indicates code should be generated to test a particular 
application. Onoe the code is generated, GUI 150 allows the user to specify that a test 
should be run usink that test code. 

It is possible thkt some requests v^ll require the coordination of multiple hardware 
elements. As will be de^scribed hereafter, one of the functions of the test engines is to 
apply a load that simulateWultiple users. In some instances, one computer can simulate 
multiple users by running nAjltiple client threads. However, there is a limit to the number 
of client threads that can run a server. 

Multiple threading the clknt test program has the advantage of being able to 
generate very large loads (tens of thousands to hundred of thousands virtual users). As 
such, multiple threading the client test program accurately emulates multiple users as far 
as the software component (EJB technology based software component) deployed on the 
application server is concerned. Only one socket is established between the host process 
and the application server and all the tMeads communicate with the software object 
through the socket. \ 

A Java Virtual Machines (JVM) is a sMtware implementation of a processor 
designed to run Java code. Each test engine can run one or more JVMs, with each JVM 
having a respective socket to the application server. \The JVM software implementation 
of a processor runs within a process. The JVM softw^e implementation of a processor 
acts as an interface between compiled Java binaV code (bytecode) and the 
microprocessor. Since each JVM software implementadon of a processor has a 
respective socket to the application server, the use of JVM soWare implementation of a 
processors very accurately emulates a large user load on the application server and on the 
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/softwardcomponents. Each test engine can run multiple JVM software implementation 
( of a procetesors, and each JVM software implementation of a processor can run multiple 
threads, xke test engines can run any mixture of multiple threads and multiple JVM 
software implementation of a processors. The test engines can also run a multithreaded 
load fi-om muUiple processes, applicable to testing EJB technology based software 
components and yOMH- (Component Object Modules). 

For distributed testing, the entire class file path specified by the user is provided 
at the remote machineV The entire class file path is compressed, also known as put into a 
jar, and the compressed Vile or jar is sent to the remote machine. In such a maimer testing 
software components witnVdistributed execution is greatly simplified. 

FIG. 4 shows the process by which queue manager 320B coordinates the actions 
of test engines located on separate servers. At step 410, queue manager 320B waits for 
the required number of test engines to become available. Once the test engines are 
available, at step 412 queue manag^ 320B sends commands to each test engine that will 
be involved in the test to dovmloadNjie test code from the appropriate one of the code 
generators 212A and 212B. 

Queue manager 320B then beginsNhe process of synchronizing the test engines 
located on different servers. Various methods could be used to synchronize the servers. 
For example, if very great accuracy is requiredV each server could be equipped with radio 
receivers that receive satellite transmissions that are intended to act as time reference 
signals, such as in a GPS system. Calibration prVesses could alternatively be used to 
determine the amount of time it takes for a commanVto reach and be processed by each 
server. Commands could then be sent to each server at times offset to make up for the 
time differences. In the preferred embodiment, a simplevprocess is used. At step 414, 
queue manager sends a message to each server to be acting as a test engine. The message 
asks that server to report the time as kept by its own internal circuitry. 
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The multiple JVM software implementation of a processors can be synchronized 
by loadingyan agent on each system the JVM software implementation of a processors 
will run, thefl the server sends a message out to the agents and requests a time check. The 
server then seflds the client code to the JVM software implementation of a processors on 
the remote machines along with the time at which the JVM software implementation of a 
processors should start executing the client code. In this maimer highly accurate and 
repeatable load gerteration is achieved. Additionally, the user can specify a time offset 
for each of the maWnes and JVM software implementation of a processors. For 
example, a first machiVe would be started; a second machine started ten seconds later, 
etc. While this process iV independent of the absolute time on each system, the machines 
could be directed to start am predefined time based on the system time. 

Additional synchronization techniques include where the server sends the client 
code to the host, the server doe^ time check, then the server sends the start time to the 
host. Altemately the server coulX send the client code and start time to the hot. Yet 
another synchronization technique mcludes having a service (e.g. JINI) nmning on the 
host computer. The server sends a message to synchronize in "X" number of ticks. The 
services wait for "X" number of ticks men executes the client code. Further, the JVM 
software implementation of a processors Oould run in an unsynchronized mode, wherein 
the JVM software implementation of a\ processors start running the client code 
independently of other JVM software implemWation of a processors 

Upon receiving the internal time as kept W each of the servers, at step 416 queue 
manager 320B adds the same offset to each localVme. At step 418, queue manager 418 
sends the offset times back to the servers. The offseKlocal times become the local starting 
time of the test for each server. Each server is instrdcted to start the test when its local 
time matches the offset local time. In this way^ all the servers start the test 
simultaneously. 


Queue manager 320B waits for the execution of the te^t code at block 420. Some 
test cases will entail multiple tests. At step 422, a check is made of whether the particular 
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test being executed requires more tests. For example, as will be described in greater 
detail bdow, one kind of test case that test system 110 will run is a load test. During a 
load testXmultiple test engines, each executing muhiple client threads, execute to 
simulate mmtiple users accessing application under test 114. An operating parameter of 
application imder test 1 14 is then measured. In the preferred embodiment, the number of 
simultaneous Users being simulated can be varied and the operating parameter measured 
again. Taking data at multiple load conditions allows test system 110 to determine the 
affect of load on anplication under test 114. Measurements of these types require that a 
full test case includk multiple repetitions of the same test with different load conditions. 

If there are mWe conditions under which the test should be run, execution 
proceeds to step 424. AtWp 424, the number of client threads is increased as needed for 
the new test case. ExecutiVi then loops back to step 410. At step 410, queue manager 
320B waits for the required Wnber of test engines to be available. When the hardware is 
available, the test is performed through steps 412, 414, 416, 418 and 420 in the same 
manner as for the first test condition. At step 422, a check is for whether the request 
entails multiple test conditions. If there are no further test conditions, the request from 
queue 318B is considered complete. If there are further test conditions that need to be 
run to complete the request, the process is again repeated. 

For test system 1 10 to operate, it\s necessary that there be test code. A user could 
provide test code. Or, test code could be provided by automatic code generation systems, 
such as TESTMASTER sold by TeradynAsoftware and System Test of Nashua, NH. 
However, FIG. 2 illustrates that code generatiirs 212A and 212B are used in the preferred 
embodiment to create the code. Tuming to FIQl 5, greater details of a code generator 212 
are shown. \ 

Code generator 212 contains several scripts. 512. Each script is a sequence of 
steps that code generator 212 must perform to create cwde that performs a certain type of 
test. The scripts can be prepared in any convenient OTOgramming language. For each 
type of test that test system 1 10 will perform, a script is wovided. User input on the type 
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n I of test tkat is desired specifies which script 512 is to be used for generating code at any 
given timev 

The selected script 512 assembles test code 516. The information needed to 
assemble test code 516 comes from several sources. One source of information is the test 
templates 514. There are some steps that are needed in almost any kind of test. For 
example, the objecKbeing tested must be deployed and some initialization sequence is 
required. If the tests are timed, there must be code that starts the test at a specified start 
time and an ending time of the test must be recorded. Also, there must be code that 
causes the required data t\ be logged during the test. After the test, there might also be 
some termination steps thatWe required. For example, w^here the initialization started 
with a request for a referenced a particular EJB technology based software component, 
the test code will likely terminate with that reference being released. The test code to 
cause these steps to be performed is captured in the set of templates 514. 

In addition, there might be different templates to ensure that the test code 516 
appropriately reflects inputs provided ftv the user. For example, different containers 
might require different command format^to achieve the same result. One way these 
different formats can be reflected in the test\ode 516 is by having different templates for 
each container. Alternatively, a user might beVble to specify the type of information that 
is to be recorded during a test. In that instance, a data logging preference might be 
implemented by having a set of templates that differ in the command lines that cause data 
to be recorded during a test. \ 

The templates are written so that certain spacesycan be filled in to customize the 
code for the specific object to be tested. In the prefernsd embodiment, code generator 
212 generates code to test a specific EJB technology basted software component in an 
application under test. One piece of information that will nteed to be filled in for many 
templates is a description of the EJB technology based softwai^ component being tested. 
Another piece of information that might be included is user co\ie to put the application 
under test in the appropriate state for a test. For example, in testtng a component of an 


- 18 - 



application that manages a database of account information for a bank, it might be 
necessaryV) have a specific account created in the database to use for test purposes or it 
might otherwise be necessary to initialize an application before testing it. The code 
needed to caufee these events might be imique to the application and will therefore be best 
inserted into tke test code by the tester testing the application. In the illustrated 
embodiment, this\code is inserted into the template and is then carried through to the final 
test code. \ 

The template might also contain spaces for a human tester to fill in other 
information, such as specific data sets to use for a test. However, in the presently 
preferred embodiment, datV sets are provided by the human user in the form of a data 
table. \ 

DataTables are used to aroly large datasets to the application under test. In order 
to properly load test software conMonents, it is necessary to specify large datasets. The 
software component is inspected ana\a template is provided. This template may be in a 
.CSV (comma separated value) formatVlthough other formats could also be used. In the 
datatable the columns are used for paraiAeters and the rows representing each user. The 
software cycles through each row of data^ as each user. If there are fewer rows than 
virtual clients, the software will go to the efad of the data set then start over beginning 
with the first row. \ 

Code generator 212 could generate functional tests. Functional tests are those 
tests that are directed at determining whether the Yean correctly performs its required 
ftmctions. In a fimctional test, the software under test \s exercised with many test cases to 
ensure that it operates correctly in every state. Data tables indicating expected outputs 
for various inputs are used to create functional test softwiare. However, in the presently 
preferred embodiment, code generator 212 primarily generates test code that performs 
load tests. In a load test, it is not necessary to stimulatAthe software under test to 
exercise every possible function and combination of ftmctions me software is intended to 
perform. Rather, it is usually sufficient to provide one test coiidition. The objective of 
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the lo^ test is to measure how operation of the software degrades as the number of 
simultan^us users of the application increases. 

In th\ preferred embodiment, test system 110 contains scripts 512 to implement 
various types of load tests. One type of load test determines response time of an EJB 
technology baseX software component. This allows the test system to vary the load on 
the EJB technology based software component and determine degradation of response 
time in response to mcreased load. Another type of load test is a regression type load 
test. In a regression type test, the script runs operations to determine whether the EJB 
technology based software component responds the same way as it did to some baseline 
stimulus. In general, the\response to the baseline stimulus represents the correct 
operation of the EJB technology based software component. Having a regression type 
test allows the test system 1 10 tt& increase the load on a bean and determine the error rate 
as a, function of load. \ 

To generate test code 516 for these types of load tests, the script 512 must create 
test code that is specific to the bean under test. The user provides information on which 
bean to test through GUI 150. In the preyed embodiment, this information is provided 
by the human tester providing the name of fee file within the application under test that 
contains the "deployment descriptor" for the specific bean under test. This information 
specifies where in the network to find the Bean under test. Script 512 uses this 
information to ascertain what test code must be generated to test the bean. 

Script 512 can generate code by using the attributes of the platform independent 
language in which the bean is written. For the example of Sun JAVA language being 
used here, each bean has an application program interface called a "reflection." More 
particularly, each bean has a "home" interface and a "remote" interface. The "home" 
interface reveals information about the methods for creating oXfinding a remote interface 
in the bean. The remote interface reveals how this code canXbe accessed from client 
software. Of particular interest in the preferred embodiment, \he home and remote 
interfaces provide the information needed to create a test program toVccess the bean. 
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^utoGen creates timer folders for each of the methods in a test. The methods are 
^ organized\by topic (for example all Getter Methods are put into the Get Method folder). 
The results Ve displayed organized in the same folder layout that AutoGen and the user 
defined in the\client program. This approach has the advantage that it is flexible and 
totally customizable by the user. Each parent folder calculates the average of the 
response time of each item within the folder and this works hierarchically. The parent 
folder may also per%m other calculations (such as total, etc.) on the items within each 
folder. 


Using the reflectiorL any program can determine what are known as the 
"properties" and "methods" oly bean. The properties of a bean describe the data types 
and attributes for a variable usedW the bean. Every variable used in the bean must have 
a property associated with it. In this way, script 512 can automatically determine what 
methods need to be exercised to test aSbean and the variables that need to be generated in 
order to provide stimulus to the methods. The variables that will be by the methods as 
they are tested can also be determined. Insthe preferred embodiment, this information is 
stored in symbol table 515. 


Symbol table 515 is a file in any convenient file format. Many formats for storing 
tabular data are known, such as .xml format. Onckthe information on the methods and 
properties are captured in a table, script 512 can useNhis information to create test code 
that exercises the methods and properties of the partiWlar component under test. In 
particular, script 512 can automatically create a variablev of the correct data type and 
assign it a value consistent with that type for any variable used in the bean. 

FIG. 5 shows a data generator 518. Data generator 5iv8 uses the information 
derived from the reflection interface to generate values for variables used during testing 
of a bean. There are many ways that appropriate values could bdgenerated for each 
variable used in the test of a particular bean. However, in the commerc^l embodiment of 
the present invention, the user is given a choice of three different algorithms that data 
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generatV 518 will use to generate data values. The user can specify "maximum," 
"minimum" or "random" If the maximvmi choice is specified, data generator 518 
analyzes thXproperty description obtained through the reflection interface and determines 
the maximuA permissible value. If the user specifies "minimum" then data generator 
518 generates me smallest value possible. If the user specifies random, data generator 
518 selects a valii^at random between the maximum and the minimum. 

In many instances where a load test is desired, the exact value of a particular 
variable is not import^t. For example, when testing whether a bean can properly store 
and retrieve a value from a database, it usually does not matter what value is stored and 
retrieved. It only matters tKat the value that is read from the database is the same one that 
was stored. Or, when timing, the operation of a particular bean, it will often not matter 
what values are input to the\method. In these scenarios, data generator 518 can 
automatically generate the valuesVor variables used in the test code. 

In cases where the specific v\lues of the variables used in a test are important, 
code generator 212 provides the user wdth another option. Rather than derive values of 
variables from data generator 518, script Vl 2 can be instructed to derive data values from 
a user provided data table 520. A user might, for example, want to provide a data table 
even for a load test when the execution time M a particular fimction would depend on the 
value of the input data. 

A data table is implemented simply as a filte on one of the computers on network 
122. The entries in the table, specifying values for\particular variables to use as inputs 
and outputs to particular methods, are separated by \ielimiters in the file. A standard 
format for such a table is "comma separated values" or CSV. In a preferred 
embodiment, test system 110 includes a file editor - Of the type using conventional 
technology - for creating and editing such a file. In addition, test system 110 would 
likely include the ability to import a file - again using knoM^ techniques - that has the 
required format. 

The methods of a bean describe the fimctions that bean c^ perform. Part of the 
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/ description of the method is the properties of the variables that are inputs or outputs to the 
methodX A second part of the description of each method - which can also be determined 
through ttte reflection interface - is the command needed to invoke this method. Because 
script 512 can determine the code needed to invoke any method and, as described above, 
can generate\data values suitable to provide as inputs to that method, script 512 can 
generate code tC) call any method in the bean. 

In the preftrred embodiment, directed at load testing, the order in which the 
methods of a bean called is not critical to an effective test. Thus, script 512 can 
automatically generate Vseful test code by invoking each method of the bean. The order 
in which the methods are\invoked does not matter if the only parameter that is measured 
is the time it takes the metHfods to execute. 

More sophisticated test\ can be automatically built by relying on the prescribed 
pattern for the language. In SxmVAVA, entity beans for controlling access to a database 
should have methods that have aVefix "set" or "get". These prefixes signal that the 
method is either to write data into ^.database or to read data from the database. The 
suffix of the method name indicates which value is to be written or read in the database. 
For example, a method named setSSN\hould perform the function of writing into a 
database a value for a parameter identified as SSN. A method named getSSN should read 
the value from the parameter named SSN. \ 

By taking advantage of these prescribed pattems, script 512 can generate code to 
exercise and verify operation of both methods. A piece of test code generated to test 
these methods would first exercise the method setSSN by providing it an argument 
created by data generator 518. Then, the method getSSN might be exercised. If the get 
method returns the same value as the argument that was supplied to the set method, then 
it can be ascertained that the database access executed as exacted. 

For many types of enterprise wide applications, theM)eans most likely to be 
sensitive to load are those that access the database. Thus, tWing only set and get 
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metho^is provides very useful load test information. 


However, the amount of testing done can be expanded where required. Some 
beans alsoVontain methods that create or find rows in a database. By convention, 
methods that\:reate or find rows in a database are named starting with "create" or "find." 
Thus, by reflecting the interface of the bean, script 512 can also determine how to test 
these methods. These methods can be exercised similarly to the set and get methods. 
The properties revealed through the application interface will described the format of 
each row in the database. Thus, when a create method is used, data can be automatically 
generated to fill that rowv thereby fiiUy exercising the create method. 

In a preferred embcKiiment, find methods are exercised using data fi-om a user 
supplied data table 520. OfterK databases have test rows inserted in them specifically for 
testing. Such a test row would likely be written into data table 520. However, it would 
also be possible to create a row, fill it with data and then exercise a find method to locate 
that row. \ 

Once the commands that exercftie the methods of an EJB technology based 
software component are created, script 512 oan also insert into the client test code 516 the 
commands that are necessary to record the oV)uts of the test. If a test is checking for 
numbers of errors, then test code 516 needs toVontain instructions that record errors in 
log 216. Likewise, if a test is measuring responsVtime, the test code 516 must contain 
instructions that write into log 216 the information\from which response time can be 
determined. In the described embodiment, all major da^base fimctions can be exercised 
with no user supplied test code. In some instances, it mi^t be possible to exercise all 
the functions with all test data automatically generated. AiUhe required information 
could be generated from just the object code of the application imder test. An important 
feature of the preferred embodiment is that it is "minimally invasiveX- meaning that very 
little is required of the user in order to conduct a test and the test abes not impact the 
customer's environment. There is no invasive test hamess. The client (wde runs exactly 
like the code a user would write. \ 


n some scenarios, it will be necessary or desirable for a user to insert specific 
steps intoStiie test code 516. Test system 110 allows for the user to edit test code 516. 
For examplL some beans will perform calculations or perform functions other than 
database acces^. In these instances, the user might chose to insert test code that exercises 
these functionsX However, because bottlenecks in the operation of many N-tiered 
applications occuXin the entity beans that manage database transactions, the simple 
techniques describecMierein are very useful. 

The results of anV tests are stored in log 216. FIG. 2 shows log 216 as a separate 
hardware element attached to network 122. Log 216 could be any type of storage 
traditionally found in a netw)rk, such as a tape or disk drive attached to a computer 
server. For simplicity, it is shoWn as a separate unit, but could easily be part of any other 
server in the network. \ 

Many types of data could be\tored in log 216. For example, there are several 
possible ways that "response time" coulkd be measured. One way is that the total time to 
execute all the methods in a bean could \)e measured. Another way is that the start up 
time of a bean could be measured. The startup time is the time it takes from when the 
bean is first accessed until the first method isvable to execute. Another way to measure 
response time is to measure the time it takes for each method to execute. As another 
variation, response time could be measured baseld on how long it takes just the "get-" 
methods to execute or just the "set-" methods to exeOute. 

Different measurements must be recorded, depending on which measure of 
response time is used. For example, if only the total re^sponse time is required, it is 
sufficient if the test code simply records the time that the V)rtion of the test code that 
exercises all the methods starts and stops executing. If th\ startup response time is 
required, then the client test code must record the time that it firs\accesses the bean under 
test and the time when the first method in the test sequence is reac^ to be called. On the 
other hand, if the response time is going to be computed for each method, the client test 



codeVmust record the time before and after it calls each method and some indication of 
the method being called must also be logged. Similar information must be recorded if 
responses of just "get-" or "set-" functions are to be measured, though the information 
needs to Be recorded for only a subset of the methods in these cases. 

In acmition, when there are multiple users being simulated, there are multiple 
values for eackdata point. For example, if test system 110 is simulating 100 users, the 
time that it takes for the bean to respond to each simulated user could be different, 
leading to up to 100 different measurements of response time. The response time for 100 
users could be presented as the maximum response time, i.e. the time it takes for all 100 
simulated users to finffeh exercising the bean under test. Altemative, the average time to 
complete could be reported as the response time. As another variation, the range of 
values could be reported. \ 

In the preferred embodiment, the client test code 516 contains the instructions that 
record all of the information th^^t would be needed for any possible measure of response 
time and every possible displaAformat. The time is recorded before and after the 
execution of every method. Also, an indication that allows the method to be identified is 
recorded in log 216. To support analysis based on factors other than delay, the actual and 
expected results of the execution of Wh method are recorded so that errors can be 
detected. Also, the occurrences of excemions are also recorded in the log. Then, data 
analyzer 218 can review the log and dispfSay the response time according to any format 
and using any definition of response time aesired. Or the data analyzer can count the 
number of exceptions or errors. \ 

Once the data is stored, the user can specif the desired format in which the data 
is to be presented. Data analyzer 218 accepts commands from the tester concerning the 
format of the output and analyzes the data appropriatd^. In a preferred embodiment, test 
system 110 has the ability to present the resuhs of tests graphically to aid the tester in 
understanding the operations - particularly performance bbttleneck - of application under 
test 114. \ 
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/ Data analyzer 218 generates output in several useful formats. One important 
cmtput is ^response time versus load graph. Log file 216 contains the starting and 
stopping times of execution tests for a particular test case. The test case includes the 
same measurements at several different load conditions (i.e. with the test engines 
214A...214C simulating different numbers of simultaneous users). Thus, data analyzer 
can read through me data in log 216 and identify results obtained at different load 
conditions. This dat^an be graphed. 

Another useful analysis is the number of errors per second that are generated as a 
function of the niunber on^imultaneous users. To perform this analysis, test code 516 
could contain instructions mat write an error message into log 216 whenever a test 
statement produces an incorrect result. In the database context, incorrect results could be 
identified when the "get" functron does not return the same value as was passed as an 
argument to the "set" function. \ Or, errors might be identified when a bean, when 
accessed, does not respond or responds with an exception condition. As above, data 
analyzer 218 can pass through the Ic^e file, reading the numbers of errors at different 
simulated load conditions. If desired, the errors can be expressed as an error count, or as 
an error rate by dividing the error count b\ the time it took for the test to run. 

Some examples of the types of outputs that might be provided are graphs 
showing: transactions per second versus numbCT of users; response time versus number of 
users; exceptions versus numbers of users; errork versus numbers of users; response time 
by method; response time versus run time and transactions per second versus run time. 
Different ways to measure response time were Miscussed above. In the preferred 
embodiment, a transaction is defined as the execution of one method, though other 
definitions are possible. \ 

Run time is defined as the total elapsed time in nWing the test case, and would 
include the time to set up the execution of EJB technology\based software components. 
Viewing response time as a function of elapsed time is usefiiL for example, in revealing 


problelms such as "memory leaks". A memory leak refers to a condition in which 
portion^of the memory on the server running the application under test gets used for 
unproducm^e things. As more memory is used unproductively, there is less memory 
available rar running the application under test and execution slows over time. 
AltemativelyVviewing results in this format might reveal that the application under test is 
effectively utilizing caching. If caching is used effectively, the execution time might 
decrease as elapsed time increases. Turning now to FIG. 6, operation of test system 110 
from the tester's perspective is illustrated. FIG.6 illustrates the tester interface to be a 
traditional web browser interface as it would appear on the terminal of PC 152 or 154. 
This type of interfac^is the result of using a web server to implement GUI 150 with 
commercially available Wowser software on the tester PC's 152 and 154. 

Element 612 is theWowser toolbar. The browser provides the ability for the 
tester to perform such functions as printing and connecting to pages presented by various 
servers in the system. The tester performs these fiinctions through the use of the tool bar 
612. Field 614 indicates the servfer to which the PC 152 or 154 is currently connected. In 
the illustrated example, field 614 contains the address on network 122 for the server 
containing GUI 150. \ 

Window 610 is the area of the screen of PC 152 or 154 where information 
specific to test system 110 is displayed. \As with traditional web based applications, this 
information is displayed as the result of HTML files that are downloaded over network 
122. Certain elements displayed in windtow 610 represent hyperlinks, which when 
accessed by a tester cause other pages to be cbvraloaded so that different information is 
displayed for the tester. Other elements displayed in window 610 represent data entry 
fields. When a human tester fills in these fields,\nformation is submitted to test system 
110. \ 

Tabs 616 represent choices a tester can select ^r operating mode. FIG. 6 shows 
three tabs. There is one tab for "setup", one for "test cask' and one for "results". Each of 
the tabs is hyperlinked. These hyperlinks connect to pagesvon servers in the network that 



are progl^ammed to manage the test system through a particular phase. Setup is for 
configurings the test system, such as by identifying addresses for servers on network 122 
that contain test engines. "Test case" is for creating and running a particular test case, 
which in the preferred embodiment means test code for a particular bean and parameters 
specifying how mat test code is to run. "Results" is for viewing the results of a test case 
that has executedX In FIG. 6, the "SETUP" tab is shovm selected, meaning that the 
balance of window VlO displays hyperlinks and fields that are appropriate for entering 
data or commands for me SETUP phase. 


Region 618 contaiils a set of hyperlinks. These hyperlinks present a list of 
choices to the user about v^at can be setup. Selecting one of these hyperlinks will 
change the information appeariiig in region 620. It is well knovm in the art of web based 
software to have hyperlinks on>9ne part of window change information appearing in 
another part of the window. 

In the Example of FIG. 6, th\ hyperlink "Machine" in region 618 has been 
selected. Therefore, region 620 contains^elds that can be used to identify a machine to 
be added to test system 110. In FIG. 6, a screen to add a client host computer is shown. 
A client host computer acts as a test engine 214A. . .214C. 


Region 618 also contains hyperlinks for VProjects" and "Data Tables". As 
described above, test system 110 can be used by rn^ltiple testers or can be used to test 
multiple applications under test. To segregate the inflation relating to various tasks, a 
"Project" is defined. All information relating to a particular project is identified with the 
project. In this way, test system 110 can identify information that is logically related. 
For example, test code developed to test a particular bean ia an application will be given 
the same project name as test code to test other beans in that application. In this way, 
general information about the application - such as the path toVhe server through which 
the application can be accessed is stored once with the project rattier than with each piece 
of test code. As another example, the test results can be associated\vith the test code that 
generated them through the use of projects. 
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The hyperlink for "Data Tables" allows the tester to identify to the system where 
data tabrfes are stored to test specific beans. In general, the tester will create the data 
tables by n^d or automatically apart from test system 1 10 based on the features that the 
tester desiresSto have tested. The data tables will be stored in files on a computer on 
network 122. Before use by test system 110, the tester must provide the network address 
for that file. \ 

Field 622 is ^onenu field. It is a drop down menu box. When accessed, it will 
display a menu of choices of the actions the tester can specify for test system 1 10. The 
contents of the menu is cqitext sensitive, meaning that for any given page, only the menu 
choices that are appropriateVre displayed. Actions that a user might want to choose from 
the menu are things such as eait, delete, add ne>v, rename, etc. 

Turning now to FIG. 7, a Screen of tester PC 152 or 154 is shovm when the TEST 
CASE tab selected. Selecting the TEST CASE tab allows the tester to specify what is to 
be tested and how the test is to be cWiducted. In the example of FIG, 7, window 610 
contains information that describes a test case. This particular page is displayed when the 
"edit" has been selected in menu field 622. Field 710 indicates the name of the project 
to which the test case is associated. FieldylO is a screen display element known as a 
drop dovra box. When activated by a tester Vield 710 will become a list of all projects 
that have been previously created by and testerVsing test system 110. As shown in FIG. 
7, a project called "Demo Project" is being used. \ 

Field 712 identifies the particular test case W name. Field 712 is also a drop 
dovm box, allowing previously defined test cases to bkselected from a list, hi addition, 
one of the options that appears when the drop down oox is accessed is "<New Test 
Case>". When selected, a new test case is created and information about this test case 
can be entered. This type of menu is common for programs with graphical user interfaces 
and is used at several points in the interface for presenting choices to a human tester. 
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/ In\the example of FIG. 7, a test case "Customer Test" has been created and 
informatiomhas already been entered. In region 714, there are a series of fields in which 
data can be entered or changed to define or modify the test case. In field 716, a name can 
be provided for\he test case. This name allows the test case to be referenced in other 
parts of the test system. 

In field 718, ^description of the test case can be entered. This description is not 
used by the automatic features of the test system, but can act as a note to a human tester 
to signify what the test cafee does or why it was created. Field 720 is likewise not used by 
the automated system. However, this field holds the name of an individual who authored 
the test case to facilitate othekadministrative functions. 

Field 722 is a "radio buttbn" type field. It presents a tester with a list of choices, 
only one of which can be selectedVt a time. In this example, field 722 allows the tester 
to specify the type of test. As previously described, code generators 212A and 212B 
contain a plurality of scripts 512 that generate test code. As described above, the script 
assembles templates and generate comiimid lines for a particular type of test that is to be 
conducted. Thus, the tester must specify\he test type in order to allow the appropriate 
script to be selected. In this example, only two types of tests are presented as options to a 
tester - a load test and a functional test. Thesfe types of tests were discussed above. Field 
724 allows the tester to specify a "deployment descriptor." In the JAVA language, every 
bean has associated with it a deployment descrip^r. The deployment descriptor is a file 
that identifies the bean and defines the parameters Vith which the EJB technology based 
software component will be deployed on the apwcations server. Examples of the 
parameters are the number of instantiations of the\EJB technology based software 
component with which to start the applications serve^ (sometimes called a "pooling 
number") and how much memory is allocated to the beanVphese functions are performed 
by the container 130. \ 

The tester provides the deployment descriptor by entWng the path to a file on 
network 122. The test system 1 10 reads the deployment descriptor to find the name of 
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the bean under test and then to access the bean through reflection to determine its 
methods Vid properties. 



Field\726 allows the tester to specify the type of data to be used in creating the 
test code 516\ As described above, in the preferred embodiment, the data can be 
automatically generated by test system 1 10 or can be supplied through a data table. For 
automatic data generation, the data can be generated by using the maximum possible 
value of each variabPe, the minimum possible value or a value randomly selected between 
the maximum and theVinimum. Alternatively, the data can be specified by a data table. 
In FIG. 7, field 726 indicates that tester desires test system 1 10 to generate data using the 
data table named dd.csv.X If the tester had wanted the test system to automatically 
generate data, the tester would specify in field 726 whether the data was to be generated 
randomly or whether the maximim or minimum values were to be used. 

Field 728 allows the user to\pecify the server on which the application under test 
runs. FIG. 1 shows that the beans 13Aof the application under test are within a container 
130. In this context, "server" refers t\ the software that creates the container for the 
application. For each platform indepenldent language, there are a limited number of 
commercially available servers. Test syste^ 110 contains script files that will generate 
the appropriate test code for any server. While most of the client test code will be server 
independent, it is possible that servers will immement certain fimctions in unique ways. 
Thus, there needs to be script files that accoun\for the functions that are implemented 
differently on certain servers. 

FIG. 8 shows a screen display when the tester has used menu field 622 and 
selected to have the deployment descriptor for a test cas\ to be displayed. If desired, the 
tester can then edit the deployment descriptor to try altemadve configurations. 


FIG. 9 shows a screen display when a tester has selected from menu field 622 to 
have the generated test code displayed. In FIG. 9, the testVode 516 is identified as 
"client code" because it will simulate the operation of a clienK 120 of the application 
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^der test 1 14. The displayed code corresponds to the code generated for the project and 
test Vase identified in fields 710 and 712. In this screen, the tester can also edit the test 
codeA 

One instance when a tester might desire to edit test code is when most of the 
commands in the test code can be automatically generated, but certain commands must 
have specmc data values for the application under test to function properly. The tester 
could have Vest system 1 10 automatically generate the test code, but then manually edit 
the few data Values that matter. As an example, a particular bean might include a method 
that processe^positive and negative values differently. The tester might know that 
processing negative numbers takes longer and therefore change a randomly generated 
value to a negativfe number. 

An altemativX scenario in which a tester might wish to edit test code 516 is when 
the bean being tested contains methods to be tested other than those that follow a pattern, 
such as the "set", "gef , Vcreate" and "find" methods described above. The tester might 
create test code that tests methods that do not follow the pattern. This code could then be 
inserted by the human testeninto the generated test code. 

FIG. 10 shows a screenMisplay for another function performed by a human tester 
while specifying the TEST CASE, also selected through menu field 622. FIG. 10 is a 
screen display used when the test case is to be run. The project and specific test case is 
specified by entries in fields 710 andv712. Information about how to run the test case is 
entered in region 1 0 1 0 . \ 

Region 1010 contains several fielck Field 1012 indicates the name of the file in 
log 216 where the results of executing the\test case will be stored. In this way, data 
analyzer 218 will be able to locate the dataVor a particular test case to analyze and 
present to a tester in the desired form. \ 

Field 1014 gives the URL - or network addkss - for the application under test. 
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\j ( This infomation could be identified by using the name for a particular machine that was 
previously s^-up by the human tester. Field 1016 gives the URL - or network address - 
for a server tovuse as a test engine. Again, the server could be identified by using the 
name for a particular machine that was previously set-up by the human tester. The screen 
displayed in FI<A 10 is used for an embodiment of the invention where all test 
simultaneously executing copies of the client test code are run on a single machine. If 
test system 110 incluoes multiple test engines and automatically schedules execution of 
test code as described i^onjunction with FIG. 4 above, then field 1016 is not required. 

Field 1018 gives theViaximum number of concurrent users to be simulated at one 
time during the execution of tHe test case. Field 1020 allows the user to specify the rate at 
which the number of simultaneot^ users v^U be simulated during a test. In the example of 
FIG. 10, the test case will beXcompleted after 100 users have been simulated 
simultaneously and the number of simultaneous users will increase by 10 each time the 
test code is run. For the examples given here, 10 copies of the client test code shovm in 
FIG. 9 will first be executed simultaneVisly. Then 20 copies of that test code will be 
executed simuhaneously. The test code wVl be repeated for 30, 40, 50, 60, 70, 80, 90 and 
100 simultaneous users before the test case iXcompleted. 

After the test case has been run, the tester can view and analyze the results. The 
human tester can access the fiinctions of the testysystem 110 that display and analyze 
results by selecting the RESULTS tab fi*om tabs 6l\6. FIG. 1 1 shows a page displayed 
when the RESULTS tab is selected. The page shown fn FIG. 1 1 is for when the tester has 
requested to see summary data through use of menu fielci 622. 

Fields 710 and 712 are also present on the page shovm in FIG. 11. This 
information is used by the test system to locate the apprVpriate data to display. In 
addition, there is a field 1110 that allows the user to enter theVame of the results file to 
display. As described in conjunction with FIG. 10, the user specifies in field 1012 the 
name of a results file to hold the results of a particular run of a testycase. The name of the 
results file for desired run of the test is entered in filed 1110. \ 
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FIgXi 1 shows that window 610 contains a region 1112 that lists the resuhs of a 
of a particular test case in summary fashion. Part of the information in region 1 1 12 is 
simply a display of information input by the tester when editing or running the test case. 
For example, tha target container, or the container 130 of application under test 114 is 
listed. The maximum number of concurrent users simulated during the test is also 
displayed. The file Containing the test data use for the run of the test case that generated 
the results is also displayed as is the deployment descriptor. These values are displayed 
for ease of use by the human tester. 


Region 1112 also intludes information that was developed by data analyzer 218 
from the data gathered for thfe specified run of the test case. In FIG. 11, the pieces of 
information that are displayed are the average response time and the maximum response 
time. As described above, as a test case runs, the start and stop times of the execution of 
the test code is recorded in log 219^ When the test code is run multiple times, each time 
simulating a different number of uteers, the start and stop time is recorded for each 
number of users. Data analyzer can Metermine the response time by simply computing 
the difference between the start and stoft times. Once values are determined, they can be 
averaged or the maximum can be identified. 

FIG. 12 shows a different page that cJm be selected by the user to see the results in 
graphical format. As above, fields 710, 712 aXd 1 1 10 allow the user to specify which run 
of a particular test case is used to create the resmts. The graphical display in FIG. 12 is a 
graph showing numbers of transactions per secooid as the dependent variable with the 
number of simultaneous users as the independeirt variable. As described above, the 
information needed to compute the data values for tins graph is stored in log 216 after the 
test case is run and data analyzer 218 can retrieve it. For creation of the graph in FIG. 12, 
transactions per second was defined as the average irumber of methods executed per 
second per user. This value is essentially the reciprocal o&response time. 

FIG. 1 3 shows a screen useful for displaying results t\a human tester in a slightly 
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different embodiment. As with FIG. 12, the screen display in FIG. 12 is accessed when 
the "RESULTS" tab is selected. Also like in FIG. 12, the page shown in FIG. 13 
includes ftelds 710, 712 and 1110 that allow the human tester to specify which results are 
to be displaVed. 

The pagfe shown in FIG. 13 includes an alternative way for the user to specify the 
format of the display. The screen in FIG. 13 includes menu fields 1310 and 1312. Menu 
field 1310 allows Nthe tester to specify the manner in which response time is to be 
calculated. In FIG. 13, a value of "total" has been selected in field 1310. As described 
above, the "total" respVise time is measured as the time from first access of a bean until 
all methods of the bean have been exercised. Other choices in menu field 1310 allow a 
tester to specify that result be displayed for different measures of response time. As 
described above, the presentW preferred embodiment can measure response time just on 
the start up time or response time for individual methods or for get- functions and set- 
functions. \ 

Field 1312 allows a user toV>ecify the format of the display. In FIG. 13, the 
display is in HiLo format. In this format, results are displayed as bars, such as bar 1316. 
Each bar spans from the fastest responseVme to the slowest response time. A tick mark 
showing the average is also included in thelJlustration of FIG. 13. Other choices in menu 
field 1312 would, for example, allow the human tester to see results in a line chart format 
as in FIG. 12 or in tabular format. \ 

Turning to FIG. 14, results in a tabular foraiat are shown. Field 1312 indicates 
that the display format of "Log File" has been selected. This format corresponds to the 
list shovm in region 1412. The list contains a column tor the name of the methods in the 
bean being tested. In this example, the data shovm for each method reveals the 
minimum, maximum and average execution time for that method. 

As described above, test system 110 measures response time at various load 
conditions. The displayed data represents response times at a particular load condition. 


Thus, to kiake the Hst, the tester must specify the load condition for which data is to be 
displayed. Vto allow this selection to be made, the page displayed in FIG. 14 contains a 
field 1410. \n this field, a user can enter the load condition for which data is to be 
displayed. In this example, the human tester has entered a value of 500, indicating that 
500 threads executing the test code were initiated in order to obtain the displayed data. 

Having desctibed the structure of test system 110 and giving examples of its 
application, several imtoortant features of the test system 1 10 can be seen. One feature is 
that information about \he performance of an application under test can be easily 
obtained, with much of the data being derived in an automated fashion. A software 
developer could use the tefet system to find particular beans that are likely to be 
performance bottlenecks in anV)plication. The developer could then rev^ite these beans 
or change their deployment descriptors. For example, one aspect of the deployment 
descriptor indicates the number o\ copies of the bean that are to be instantiated within 
application under test 1 14. The devteloper could increase the number of instantiations of 
a bean if that bean is the bottleneck. \ 

The test system described herein ^provides an easy and accurate tool to test EJB 
technology based software components forVcalability. It creates a user specified number 
of virtual users that call the EJB technology based software component while it is 
deployed on the applications server. The tool Moes this by inspecting the EJB technology 
based software component under test and autorn^atically generating a client test program, 
using either rules based data or data supplied by an a human tester, and then 
multithreading the client test program to drive \he EJB technology based software 
component under test. The result is a series of ^phs reporting on the performance 
versus the number of users, which provide usefiil information in an easy to use format. 

Another feature of the invention is that the tests ar^run without requiring changes 
in the application under test or even the installation of spfljcial test agents on the server 
containing the software under test. The generated test code 51 6 exercises the bean in the 
application under test using remote procedure calls. \ 
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Another feature of the described embodiment of test system 110 is that it is 
caiable. To inWase the nximber of tests that could simultaneously be run or the size of 
the tests that comd be run, more test engines could be added. Likewise, more code 
generators could b^ added to support the simulation of a larger number of simultaneous 
users. The specifm number of copies of each component is not important to the 
invention. The actuaKnumber of each component in any given embodiment is likely to 
vary from installation \p installation. The more users an application is intended to 
support, the more test engines are likely to be required. 

Another feature of tr^ described embodiment is that testing is done on the 
simplest construct in the application under test - the beans in the illustrated example. 
There are two benefits to this approach. First, it allows tests to be generated very simply, 
with minimal human intervention. Second, it allows a software developer to focus in on 
the point of the software that need\ to be changed or adjusted in order to improve 
performance. 

It should be appreciated that disp%s of specific kinds of information have been 
described. Various other analyses might baperformed. It was described that response 
times and error rates as a ftmction of load couM be graphed for display to a human tester 
for fiirther analysis. One enhancement that might be made to test system 1 10 is that the 
data analyzer 218 could be programmed to perform further analysis. It has been 
recognized that, as the load increases, there is ofteA some point at which the performance 
of the system drastically changes. In some instanceV the time to complete a transaction 
drastically increases. A drastic increase in transaction processing time indicates that the 
system was not able to effectively handle the load. Hbwever, a decrease in processing 
time can also indicate the load limit was reached. SomWimes, a system under test will 
respond with an error message more quickly than it womld take to generate a correct 
response. Thus, if the only parameter being tracked is rWonse time, a decrease in 
processing time as a ftmction of load can also signal that th^ maximum load has been 
exceeded. Of course, an increase in errors or error rate can also>signal that the maximum 
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load Vas exceeded. Data analyzer 218 could be used to identify automatically a 
maxiniVm load for a particular test case. By running multiple test cases, each test case 
focusing\on a different bean, test system 110 could automatically determine the bean that 
is the perfbrmance bottleneck and could also assign a load rating to application under test 
114. \ 

Havinft described one embodiment, numerous alternative embodiments or 
variations might be made. For example, it was described that test system 110 
automatically generates test code to exercise beans that follow a pattem for database 
access. These beaXs are sometimes called "entity beans." In general, there will be other 
beans in an applicatron that perform computations on the data or that control the timing 
of the execution of thJi entity beans. These beans are sometimes called "session beans." 
Session beans are lessXikely to follow prescribed programming patterns that make the 
generation of test code for entity beans simple. As a result, the automatically generated 
test code for session besms might not fully test those beans. In the described 
embodiment, it is expected that the human tester supply test code to test session beans 
where the automatically generked tests are inadequate. 

One possible modificatiomto the described embodiment is that the completeness 
of tests for session beans might beVncreased. One way to increase the accuracy of tests 
for session beans would be to captme data about the execution of those beans during 
actual operation of the application under test 114. This data could allow an automated 
system to determine things like appropWte data values, which might then be used to 
build a data table. Or, the captured data cbuld allow the automated system to determine 
the order in which a session bean accesses cmer session beans or entity beans to create a 
realistic test. \ 

Also, as described, test code is generate^ to test a particular bean, which is a 
simple construct or "component" of the application under test. The testing could focus 
on different constructs, such as specific methods in aV)ean. Test code could be generated 
to test specific methods within beans. Or, it was described that the system records start 
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/and atop time of the execution of the test code. The times of other events could be 
/ recorded instead or in addition. For example, start and stop times of individual methods 
might bd recorded, allowing performance of individual methods to be determined. 

Alternatively, the complexity of the constructs being tested could be increased. 
Multiple bekns might be tested simultaneously to determine interactions between beans. 
For example,Vnultiple test cases might be executed at the same time, with one test case 
exercising a sfDCcified instances of one bean and a different test case exercising a 
specified numbeXof instances of a second bean. 

As another example of a possible variation, it was described that a human tester 
can insert code into a template to do such things as put the application under test into a 
predictable state. Linles of code might be inserted directly, for example by the user 
simply typing the lines of code. Or, the tester might insert a "tag" into the template. The 
tag would identify a codeVegment stored elsewhere within the test system. In this way, 
the same code segment could be included at multiple places in the template or in multiple 
templates. \ 

As another example of possible variations, the number of templates used to 
construct test code might be varie^. One possibility is that each template contains all of 
the steps needed to initialize, run and terminate a test. Thus, test code would be created 
by filling in a single template. Alternatively, each template might contain only the steps 
needed to perform one function, such\as initialization, testing or termination. In this 
implementation, test code would be created by stringing together multiple templates. 

Also, it was described that in nmning\a test that a number of simultaneous users is 
"synchronized". Simultaneous users are sinmlated by synchronizing copies of the test 
code on different servers and on the same serveX The term "synchronized" should not be 
interpreted in so limited a way as to imply tha\ multiple copies are each performing 
exactly the same function at exactly the same timV Thus, when described herein that 
execution is synchronized, all that is required is that each copy of the code is making 
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requetets of the application under test during the window of time when the test is being 
executed. Some copies of the code will likely start execution sooner or end sooner than 
the othqrs. However, as long as there is overlap in the timing of execution, the test 
programs\can be said to be synchronized or running concurrently. 


As aWther variation, it was described that the test system 110 provides outputs 
indicating theVierformance of an application under test as a function of load. These 
outputs in grapmcal or tabular form can be used by an application developer to identify a 
number of concurrent users at which problems with the application are likely to be 
encountered. Potdntial problems are manifested in various ways, such as by a sudden 
change in response tmie or error rate as a function of load. Test system 1 1 0 could readily 
be programmed to autWatically identify patterns in the output indicating these problem 
points. \ 

Another useful modification would allow test system 110 to aid in identifying 
settings for various parametesrs in the deployment descriptor. As described above, the 
deployment descriptor for a osan identifies parameters such as memory usage and a 
"pooling number" indicating th\ number of instances of a bean that are created at the 
initialization of an application. These and other settings in the deployment descriptor 
might have an impact on the perfcmnance time and maximum load that an application 
could handle. One use of the test systW described above is that it allows a test case to be 
repeated for different settings in the demoyment descriptor. A human tester can analyze 
changes in performance for different settings in the deployment descriptor. However, 
test system 1 10 could be programmed to aVtomatically edit the deployment descriptor of 
a bean by changing parameters affecting pooling or memory usage. Test system 110 
could then automatically gather and present Wa showing the impact of a deployment 
descriptor on performance of an application. \ 

Even higher levels of automation could be achieved by test system 110. For 
example, test system 1 10 might test the beans in an application and analyze the results of 
testing each bean. Test system 110 might identiW the bean or beans that reflect 
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/ periormance bottlenecks (i.e. that exhibited unacceptable response times for the lowest 
numbers of simultaneous users). Then, test system 110 could run tests on those beans to 
find seftings in the deployment descriptors that would balance the performance of the 
beans inXthe application (i.e. to adaptively adjust the settings in the deployment 
descriptors\o that the bottleneck beans performed no worse than other beans.) 

It shoufd also be appreciated that computer technology is rapidly evolving and 
improved or enlwmced versions of the hardware and software components making up the 
application under \est and the test system are likely to become available. It should also 
be appreciated that me description of one device in a class is intended to be illustrative 
rather than limiting am that other devices within the same class might be substituted with 
ordinary skill in the ark For example, the application under test was described in the 
context of a conventionalVapplication accessed through a client on a PC using a web 
browser as a graphical user Vterface. It should be appreciated, though, that if the clients 
are intended to be household appliances with micro-controllers, a different interface 
might be readily substituted for the graphical user interface. 

Also, it was described that FIG. 1 1 shows summary data of a test after execution 
is completed. It will be appreciated, mough, that data on a test case execution might be 
displayed to a human tester while a tesms in process. For example, the summary screen 
might contain a field that shows the percentage of the test case that is completed. This 
value would update as the test runs. Likewise, the values for average and maximum 
response time could be updated as the data isVathered. 

Also, it was described that the objects Ijeing tested are EJB technology based 
software components, which are written in the Java language. The same techniques are 
equally applicable to applications having componems implemented in other languages. 
For example, applications written according to the OOM standard might be written in 
Visual Basic and applications written for the CORBA standard might be written in C-f-+. 

Regardless of the specific language used, these standards are intended to allow 
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/separately Meveloped components to operate together. Thus, each must provide a 
mechanism mr other applications, such as test system 1 10, to determine how to access the 
methods and properties of their components. However, there could be differences in the 
specific commands used to access components. 


In one embooiment, code generator 212 is implemented in a way that will make it 
easy to modify for geiierating test code for applications written in a different language. 
Specifically, code geneWor 212 stores intermediate results as a symbol table that is 
I independent of the specific language used to program the application imder test. The 
symbol table lists methodsNand properties for the component being tested. When to 
access these methods and wha^data to use for a particular test and what kinds of data to 
record can be detemiined from me information in the symbol table and input from the 
user. Thus, much of the functioning of code generator 212 is independent of the specific 
language used to implement the application under test. 

In this way, the language specific aspects of code generator 212 are easily 
segregated and represent a relatively small part of the code generator 212. In particular, 
language specific information is needed to access the application under test to derive the 
information for the symbol table. Language sbecific information is also needed to format 
the generated client test code. But, it is intendted that these parts of code generator 212 
could be replaced to allow test system 1 10 to testVpplications written in other languages. 
Also, it is possible that test system 110 will contain multiple versions of the language 
specific parts and the user could specify as an input tH^ language of the application under 
test. 

Therefore, the invention^should be limited only by\he spirit and scope of the 
appended claims. 


Delete the paragraphs beginning at page 42. line 1 through' page 57. line 35. 
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